Enhancing Efficiency and Workflow of Bio Statistics Teams

Hello there!

• Background: Computer Science Engineer and MBA in Business Analytics.
• Curious to learn.
• Using R and R Shiny for 6+ years.
• Architected, built and managed more than 40+ R Shiny applications and dashboards.
• Built and led teams of 10+ R developers.
• Trained 30+ colleagues on R from diverse backgrounds.
• Have extensive hands on experience on all flavors of R products - Shiny, R Markdowns, Quarto, and Package development.
• Working with ProCogia; an Official full service partner with Posit.

Current Challenges

Clinical Trails are often complex and time consuming with repeated similar workflow steps for each new variation of the trail.

Solution

Empowering Bio Statisticians with R Shiny Suite of applications

How does the journey look like to make this a success?

Migration Process Steps

• Define Scope of each process.
• Analyze repetition rate, importance, and time investment for each work request.
• Identify the most time-consuming, yet simplest workflow.
• Develop a MVP (Minimum Viable Product)
  • Showcase a demo with the smallest workflow.
  • This aids leaders in visualizing the impact of approval.
• Integrate workflows incrementally from small to large.

Engage with the Bio Stats Team

• Weekly connects with Bio Stats to ensure that the MVP build is on the right track.
• Adopt Agile methodology for faster iterative improvement and development.
• Actively listen to user (Bio Stats) feedback and take “notes”.
  • Prioritize features based on complexity, time, effort and need.
  • Pro-actively research and pick up features from the end of the task list in free time to keep developers engaged and happy.
• Utilize GitHub Issues to document feature requests or bug fixes.
  • A picture speaks a thousand words. If it can be explained with a screenshot, don’t shy away.
• Ensure commits are linked with respective issues for traceability during testing.
• Establish clear requirements to expedite implementation, with developer input on the overall app workflow.

Principles of Application Design

• Think about creating and showcasing dashboard outlines in the early discussions.
  • Prototype the UI layout using draw.io or similar tools.
  • Use conditional panels, modules, and well-designed layouts to create a user-friendly interface.
  • Iterate rapidly to test how different implementations enhance UI/UX.
• Establish a standard application/dashboard template layout such as Shiny Dashboard, bslib and/or bs4Dash.
  • Create custom R function wrappers based on organization theme and color layout.
  • Re-use the template for new builds and ideas to maintain consistency and coherence.

Principles of Application Design

• Define the flow of reactivity for the overall application/dashboard.
  • One of my favorites is to use the Waterfall Reactivity Model.
  • Helps to avoid Reactivity Spaghetti Mess
  • Avoid cluttered and confusing reactivity paths.
  • Leverage Shiny’s reactivity to update UI elements dynamically based on user input. This enhances interactivity and responsiveness.
• Provide default selections for input widgets, enabling users to explore and understand the application easily.
• Prioritize user-friendliness; if it’s not intuitive, it won’t be used.
• Incorporate “tooltips”, “notifications”, section write-ups and clear instructions to guide users in using the application.

Organize your project structure

Streamline Workflow

• Use RStudio Projects
  • Isolates your development environment.
  • Relative path referencing for your project files.
  • Segregate development flow and context.
• Version Control with Git
  • Track changes, collaborate with team members, and manage project history effectively using Git.
  • Use Issues, Pull Requests and Connected Commits for efficient and effortless parallel development.

Shiny Development Tips

• Establish and adhere to a standard file and code structure.
• Implement coding standards to facilitate seamless collaboration between Bio Statistics teams and R developers.
• Emphasize the use of functions and shiny modules whenever possible.
• Optimize Server logic via Profiling and performance tuning
  • Minimize unnecessary computations and avoid redundant calculations.
  • Use tools like profvis to identify performance bottlenecks and Optimize critical sections of your code for speed.
  • Use benchmarking techniques to compare computation speeds of various packages. Refer R function rbenchmark::benchmark().
• Create smaller, independent Proof of Concepts (POCs) for new feature requests.
  • For example, capture simulation attributes in a table and select them for subsequent runs.

Shiny Development Tips

• Debugging like a pro
  • Debug via browser(), debug(function_name), and debug points.
  • Set breakpoints, step through code, and inspect variables to troubleshoot efficiently.
• Apply standard software development principles:
  • DRY (Don’t Repeat Yourself)
  • YAGNI (You aren’t going to need it)
  • KISS (Keep it Simple Silly)
  • Document your code for humans.
  • Prioritize clean code at all times.
  • Separation of Concerns.

Facilitate Consistent Reporting

Ensuring Reproducibility

• Leverage package management tools
  • Utilize renv or packrat to manage package dependencies.
  • Provides a controlled environment for your project.
  • Ensure reproducibility and minimize version conflicts.
• Rely on packages published and actively maintained on CRAN for a validated R environment.
• Establish unit tests for every Bio Stats function used to ensure accurate and consistent results.
• Invest time to create an in-depth GitHub ReadMe with the sections Project Overview, Getting Started, User Application Flow and Usage, Key Programming Concept Implemented (if any) and Developer Guide providing comprehensive project reproducibility instructions.

Enhancing Scalability

• Implement modularization and functional programming for a plug-and-play development format across applications.
• Enable multiple studies to be added concurrently using standard git branching strategies, involving multiple concurrent developers.
• Async Programming: Evaluate longer simulations in a separate R process to keep the current R process running the shiny application free for use, preventing app performance issues.
  • Few R packages to aid this are: callr, mirai, crew, coro, future and promises
• Take a step further and deploy simulation functions as internal APIs with Plumber.
• Write your custom JavaScript and R bindings for implementing unique feature requests.
• Approach feature requests as a blend of web development, software engineering, and R development.

Utilize Automated Testing

Common Application Features

• Download visualization from any part of the application.
• Access downloadable interim simulation calculations for each process, aiding in simulation fine-tuning.
• Download dynamically rendered MS Word report with FDA submission format for faster review and iterative changes.
• Add and execute multiple simulations simultaneously for efficiency.
  • Compare similar graphs side by side for comprehensive analysis across all simulations.
  • Generate dynamic grouped plots based on the sequence of simulations requested.
• Receive email notifications to stay updated on the progress of lengthy simulations, along with attached reports.

Common Application Features

• Include and display the underlying mathematical equation for a study or simulation on the user interface using LaTeX.
• Receive appropriate notifications upon completion of each process.
• Helper Tabs for - Application Information, Usage Manual, Release Tabs, Feedback and Contact Business Lead - all through the application.
  • Distribute an updated User Manual in the quarterly release email and include it as a ‘User Manual’ tab within the application.
  • Feature a ‘User Feedback’ tab for direct communication of feedback with the project manager via email.
  • Include a ‘Release Notes’ tab to display app changes over time, promoting transparency.

Enhancing User Adoption

• Create GIFs showcasing the application layout and user flow.
  • Include them in the Git readme and announcement emails.
• Create detailed application interaction user manuals with screenshots and highlights for each step.
• Conduct regular (quarterly) training sessions to provide guidance, answer questions, and assist users with new features.
  • Record and share them for easier re-visit.
• Continuously engage user base for better ROI and on boarding.
• Prioritize most requested user features for each sprint.

Thank you

• Quarto presentation code available on GitHub at http://bit.ly/3ZHZTNf
• Slides available on GitHub Pages at https://bit.ly/3RsdExn
• Connect and/or send me a DM for a follow up question or catch up
  • Linkedin: mayank-agrawal-7jan
  • X (previously Twitter): mayank7jan
  • Mastodon: mayank7j

References - R Packages

• shinyDashboard, bslib, bs4dash for standard dashboard template.
• rmarkdown and Quarto for parameterized reporting.
• renv for package management in a R project.
• glue for interpreted string literals for dynamic reporting.
• callr for separate r sessions.
• plumber for API creation.
• httr2 for API calls.
• pins for shareable secured publishing of data, models, and R objects
• testthat, shinytest2 and shinyloadtest for testing.
• dplyr for data manipulation.
• ggplot2, plotly and echarts4r for visualization.
• profvis for code profiling and time estimation